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Preface 


This  report  details  the  work  done  by  Honeywell  Aerospace  Advanced  Technology  to 
address  two  key  challenges  in  the  transition  of  Augmented  Cognition  (AugCog) 
technology  to  the  Army’s  Future  Force:  1)  integration  of  the  Honeywell  AugCog  system 
into  the  Future  Force  Warrior  (FFW)  soldier  system  and  2)  deployment  and  evaluation  of 
the  dry  electrodes  for  the  collection  of  electroencephalogram  (EEG)  data  under  the 
helmet  in  an  operational  environment.  Separate  evaluations  were  conducted  to  test  the 
success  of  both  efforts. 

This  work  was  performed  under  contract  number  W91 1QY-07-C-0037  to  the  Natick 
Soldier  Research,  Development  and  Engineering  Center  (NSRDEC)  during  the  period 
February  2007  to  May  2008.  Under  this  contract  Honeywell  also  worked  with  NSRDEC 
to  develop  an  AugCog  Concept  of  Operations  (CONOPS).  The  AugCog  CONOPS  was 
developed  to  further  explore  the  potential  application  and  utility  of  cognitive  state 
information  by  Army  commanders  in  the  field. 
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Executive  Summary 


Honeywell  Aerospace  Advanced  Technology  addressed  two  key  challenges  in  the 
transition  of  Augmented  Cognition  (AugCog)  technology  to  the  Army’s  Future  Force: 
integration  and  deployability.  Specifically,  efforts  focused  on  (1)  integration  of  the 
Honeywell  AugCog  system  into  the  Future  Force  Warrior  (FFW)  soldier  system,  and  (2) 
deployment  and  evaluation  of  the  dry  electrodes  for  the  collection  of 
electroencephalogram  (EEG)  data  under  the  helmet  in  an  operational  environment. 
Separate  evaluations  were  conducted  to  test  the  success  of  both  efforts. 

The  Honeywell  team  integrated  AugCog  hardware  and  software  with  FFW  leader 
systems  during  the  U.S.  Army’s  Command,  Control,  Communication,  Computers, 
Intelligence,  Surveillance,  and  Reconnaissance  (C4ISR)  2007  On-The-Move  (OTM) 
Event  at  FT  Dix,  NJ.  The  team  then  collected  electrocardiogram  (ECG)  and  EEG  data  on 
the  3rd  squad  leader  during  force-on-force  exercises.  The  primary  goal  was  to 
demonstrate  the  capability  of  the  AugCog  hardware  and  software  system  to  successfully 
integrate  and  execute  within  the  FFW  Leader  System  Software.  Honeywell  demonstrated 
a  successful  integration  and  achieved  a  secondary  goal  of  collecting  ECG  and  EEG 
signals  of  a  quality  sufficient  to  classify  cognitive  state  classification  in  a  harsh 
operational  environment.  Based  on  EEG  alone,  classification  of  workload,  in  the  field, 
exceeded  80%  accuracy. 

The  secondary  goal  of  this  project  was  to  advance  the  deployability  of  EEG  sensor 
systems.  Toward  that  goal,  Honeywell  worked  with  QUASAR  to  integrate  their  dry  EEG 
electrodes  under  a  combat  helmet,  coordinated  a  field  test  at  the  Aberdeen  Test  Center, 
and  conducted  post  hoc  classification  analyses  on  the  collected  EEG  signals.  Together 
these  successes  represent  an  advance  in  integration  maturity  of  the  AugCog  system  as  it 
moves  toward  transition  into  the  Army’s  future  systems. 

In  addition,  Honeywell  has  worked  with  the  Natick  Soldier  Research,  Development  and 
Engineering  Center  (NSRDEC)  to  develop  an  AugCog  Concept  of  Operations 
(CONOPS).  The  AugCog  CONOPS  was  developed  to  further  explore  the  potential 
application  and  utility  of  cognitive  state  information  by  Army  commanders  in  the  field. 

The  proposed  AugCog  System  will  benefit  joint  human-automation  performance  due  to 
the  following:  shortened  decision-making  cycle  time,  decreased  task  time,  increased 
accuracy  of  multitask  performance,  optimized  workload  to  avoid  periods  of  low 
involvement  and  extreme  engagement  by  subordinates,  reduced  cognitive  fatigue  and 
stress,  and  increased  support  for  human  involvement  at  the  appropriate  level  of  detail 
needed  for  optimal  performance.  Development  of  these  technologies  will  provide  new 
capabilities  that  are  required  to  realize  the  goal  of  closing  the  loop  in  military  operational 
decision-making  for  the  dismounted  soldier. 

The  success  at  FT  Dix  represents  an  advance  in  integration  maturity  for  AugCog  since  the 
system  was  previously  tested  at  the  Aberdeen  Test  Center  in  2006.  During  that  test,  the  EEG 


IX 


and  ECG  signals  were  transmitted  wirelessly  to  an  off-the-body  system;  during  OTM  2007, 
the  sensing  hardware  and  signal  processing  software  were  hosted  on  the  solider-bome  FFW 
Leader  System.  Likewise,  fielding  dry  EEG  electrodes  was  another  step  toward  improving  the 
deployability  of  a  sensor-based  cognitive  state  classification  system  such  as  the  Honeywell 
AugCog  system  described  in  this  report. 
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AUGMENTED  COGNITION  TRANSITION 


1  Introduction 


The  aim  of  Augmented  Cognition  (AugCog)  research  is  to  use  physiological  and 
neurophysiological  sensors  to  detect,  in  real  time,  cognitive  states  where  cognitive 
resources  may  be  inadequate  to  cope  with  mission-relevant  demands.  Human 
performance  as  a  function  of  workload  fluctuates  subject  to  fatigue,  stress,  overload,  or 
boredom.  Efforts  have  focused  on  ways  to  leverage  cognitive  state  information  to  drive 
adaptive  systems  to  manage  information  flow  when  detected  human  cognitive  resources 
may  be  inadequate  for  the  tasks  at  hand. 

Work  in  the  field  of  AugCog  began  by  classifying  aspects  of  cognitive  processing 
(attention,  working  memory,  executive  function,  and  sensory  processing)  with  well- 
defined,  well-understood  laboratory  tasks  (often  referred  to  informally  as  “Psych  101” 
tasks).  Over  the  past  five  years,  researchers  have  moved  from  the  laboratory  environment 
to  the  field  environment,  introducing  the  artifacts  (motion,  electrical,  networking  traffic, 
and  disconnect)  and  stressors  (information  overload,  physical  load,  competition,  and 
threat  of  pain)  inherent  in  some  operational  environments  to  which  AugCog  systems 
would  be  transitioned.  The  move  from  the  laboratory  to  mobile  field  environments  brings 
several  unique  challenges  that  must  be  addressed  if  cognitive  state  assessment  is  to  be 
used  successfully  during  mobile  tasks.  Tough  sacrifices  need  to  be  made,  with  limitations 
on  the  sensors  to  be  used,  processing  power,  and  knowledge  of  the  task  environment 

From  the  start  of  the  program,  the  Honeywell  AugCog  team  worked  closely  with  the  U.S. 
Army  to  address  the  problem  of  information  overload,  expected  to  occur  with  the 
deployment  of  future  Command,  Control,  Communications,  Computers,  Intelligence, 
Surveillance,  and  Reconnaissance  (C4ISR)  technologies.  The  Honeywell  team  has 
worked  to  apply  AugCog  technologies  to  the  Future  Force  Warrior  (FFW)  domain  of  the 
individual  soldier.  FFW  will  require  an  AugCog  capability  to  manage  information 
overload  during  operational  task  performance — because  soldiers  within  future  units  of 
action  will  be  required  to  process  more  information  afforded  by  C4ISR,  assume  more 
complicated  decision-making  responsibility  (e.g.,  calling  in  netted  fires),  and  manage 
multiple  organic  autonomous  assets  (e.g.,  armed  unmanned  air  vehicle,  robotic  mule) 
within  fast-paced,  dynamic  military  operations  in  urban  terrain  (MOUT). 

Current  work  has  focused  on  transition  challenges,  in  addition  to  developing  wireless  and 
wearable  components  for  sensing  soldier  states  and  supporting  interactions  with 
information  devices,  supporting  mobile  users  in  physically  taxing  environments,  enabling 
multitasking  in  an  environment  where  mission  and  task  priorities  can  change  frequently 
and  rapidly,  detecting  and  managing  information  overload  conditions,  and  supporting 
complex  decision-making  responsibilities.  To  realize  both  the  AugCog  and  FFW 
Advanced  Technology  Demonstration  (ATD)  goals,  Honeywell  addressed  two  principal 
objectives  to  move  forward  on  the  path  of  transition:  integration  and  deployability. 
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Specifically,  efforts  focused  on  (1)  integration  of  the  Honeywell  AugCog  system  into  the 
FFW  soldier  system,  and  (2)  deployment  and  evaluation  of  the  QUASAR  dry  electrodes 
for  the  collection  of  EEG  under  the  helmet.  Separate  evaluations  were  conducted  to 
evaluate  the  success  of  both  of  these  objectives. 

In  addition,  Honeywell  has  worked  with  the  Natick  Soldier  Research,  Development  and 
Engineering  Center  (NSRDEC)  to  develop  an  AugCog  Concept  of  Operations 
(CONOPS).  The  AugCog  CONOPS  (see  Appendix  B)  was  developed  to  further  explore 
the  potential  application  and  utility  of  cognitive  state  information  by  Army  commanders 
in  the  field. 

The  proposed  AugCog  system  will  benefit  joint  human-automation  performance  due  to 
the  following:  shortened  decision-making  cycle  time,  decreased  task  time,  increased 
accuracy  of  multitask  performance,  optimized  workload  to  avoid  periods  of  low 
involvement  and  extreme  engagement  by  subordinates,  reduced  cognitive  fatigue  and 
stress,  and  increased  support  for  human  involvement  at  the  appropriate  level  of  detail 
needed  for  optimal  performance. 

Development  of  these  technologies  will  provide  new  capabilities  that  are  required  to 
realize  the  goal  of  closing  the  loop  in  military  operational  decision-making  for  the 
dismounted  soldier. 

Section  2  of  this  report  describes  the  generalized  Honeywell  AugCog  system  architecture. 
Section  3  describes  how  that  system  was  integrated  into  the  FFW  Early  Increment  Leader 
System.  Section  4  describes  the  evaluation  of  the  integrated  system  at  the  On-The-  Move 
(OTM)  technology  demonstration.  Section  5  describes  the  integration  and  evaluation  of 
dry  electrodes  technology.  Section  6  discusses  next  steps. 
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2  System  Description 


2. 1  General  Architecture 

The  Honeywell  closed-loop  integrated  prototype  (CLIP)  is  depicted  in  Figure  1 . 
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Figure  1.  CLIP  demonstration  architecture. 


Physiological  and  neurophysiological  sensors  collected  real-time  data.  These  raw  signals 
were  processed  to  remove  artifacts  due  to  noise,  motion,  and  the  environment.  After 
signal  processing,  the  clean  signals  were  classified  into  levels  of  cognitive  states  of 
interest.  Honeywell  has  focused  on  classifying  cognitive  workload.  Cognitive  workload 
of  the  soldier  was  sent  over  a  communications  network  to  be  displayed  on  the 
Commander’s  Display.  The  commander  then  determined  if  any  mitigation  was  needed. 

2.1.1  Sensor  Hardware 

Each  participant  was  outfitted  with  an  Advanced  Brain  Monitoring,  Inc.  (ABM)  EEG 
system,  a  Vital  Signs  Detection  System  (VSDS)  for  cardiac  data,  and  a  wireless 
microphone  for  recording  ambient  noise  and  verbal  communications. 

2.1.1. 1  ABM  EEG  System 

EEG  data  were  collected  from  the  ABM  EEG  sensor  headset  (Figure  2).  The  sensor 
headset  acquired  six  channels  of  EEG  using  a  bipolar  montage.  Differential  EEG  were 
sampled  from  bipolar  channels  CzPOz,  FzPOz,  F3Cz,  F3F4,  FzC3,  and  C3C4  at  256 
samples  per  second  with  a  bandpass  from  0.5  and  65  Hz  (at  3-dB  attenuation)  obtained 
digitally  with  Sigma-Delta  analog-to-digital  (A/D)  converters.  Data  were  transmitted 
across  a  Bluetooth  radio  frequency  (RF)  link  to  the  collection  laptop  via  an  RS-232 
interface. 
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Figure  2.  ABM’s  wireless  EEG  sensor  headset. 


The  sensor  headset  was  developed  by  ABM  as  a  portable  system  to  record  EEG  signals. 
The  headset  fit  snugly  on  the  head  and  housed  EEG  sensors  like  many  Food  and  Drug 
Administration  (FDA)  approved  laboratory  EEG  systems,  such  as  the  Quick-Cap  by 
Neuromedical  Supplies  or  the  Electro-Cap  by  Electro-Cap  International.  Physiological 
recordings  were  made  with  an  experimental  seven-channel  digital  physiological  recorder 
with  low-powered  EEG  and  electrooculogram  (EOG)  amplifiers  designed  specifically  for 
ambulatory  recordings.  The  analog  box  included  input  jacks  for  the  electrode  leads  that 
carried  event  markers,  an  on/off  switch,  amplifiers,  and  optical  isolation  designed  to  meet 
UL544  requirements.  The  analog  box  was  coupled  to  a  Real  Time  Devices 
microcomputer  (commercially  available  model  DSi486SLC,  State  College, 

Pennsylvania),  which  provides  A/D  conversion,  operates  the  data  acquisition  software, 
and  stores  the  data  to  hard  drive. 

2. 1.1. 2  Hidalgo  Vital  Signs  Detection  System  (VSDS) 

The  VSDS  (Figure  3)  measured  heart  rate,  respiration  rate,  and  body  motion  and  position. 
The  VSDS  (Bluetooth-enabled  version)  came  with  a  Bluetooth  (Mini  Mitter  Co.  and 
Hidalgo  Ltd.)  radio  that  operated  in  full  disclosure  mode.  In  this  mode,  both  waveform 
and  summary  data  were  transmitted  across  a  Bluetooth  communications  link.  The 
AugCog  system  utilized  the  ECG  waveform  (two  views,  sampled  at  256  Hz)  and  the 
three-axis  accelerometry  waveform  (sampled  at  25.6  Hz)  signals. 
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Figure  3.  Hidalgo  Vital  Signs  Detection  System  (VSDS). 


2.1.2  Signal  Processing 

Conducting  military  maneuvers  in  operational  environments  (e.g.,  urban  terrain)  often 
does  not  allow  an  individual  to  remain  stationary  and  can  demand  simultaneous  cognitive 
and  physical  activity.  Inferring  cognitive  state  from  noninvasive  neurophysiological 
sensors  is  a  challenging  task  even  in  pristine  laboratory  environments.  Artifacts  ranging 
from  eye  blinks  to  muscle  artifacts  and  electrical  line  noise  can  mask  the  subtle  electrical 
signals  associated  with  cognitive  functions  of  interest.  These  concerns  were  particularly 
pronounced  in  the  context  of  the  dismounted  ambulatory  soldier.  For  instance,  difficulties 
related  to  processing  of  EEG  signals  in  real-world  settings  include  factors  associated  with 
both  participant  motion  and  the  operational  environment  itself.  Specifically,  artifacts 
related  to  participant  motion  include  high-frequency  muscle  activity,  verbal  communica¬ 
tion,  and  ocular  artifacts  (consisting  of  eye  movements  and  blinks),  whereas  artifacts 
related  to  the  operational  environment  include  electrical  noise  that  creates  interference 
with  the  EEG  signal.  Thus,  utilization  of  research  methods  involving  EEG  in  operational 
environments  necessitates  the  use  of  real-time  algorithms  for  signal  detection  and 
removal  of  artifacts.  Although  real-time  signal  processing  and  classification  of  the  EEG 
has  been  implemented  previously,  the  Honeywell  team  was  the  first  to  realize  this 
classification  in  a  truly  mobile,  ambulatory  environment  with  dismounted  soldiers 
(Domeich  et  al.,  2007). 

The  ABM  system  supported  an  independent  signal  processing  stream.  Quantification  of 
the  EEG  in  real  time  was  achieved  using  signal  analysis  techniques  that  identified  and 
decontaminated  eye  blinks  and  identified  and  rejected  data  points  contaminated  with 
electromyographic  (EMG)  artifacts,  amplifier  saturation,  and/or  excursions  due  to 
movement  artifacts  (see  Berka,  Levendowski,  Cvetinovic,  Petrovic  et  al.,  2004,  for  a 
detailed  description  of  the  artifact  decontamination  procedures).  Decontaminated  EEG 
was  then  segmented  into  overlapping  256-data-point  windows  called  overlays.  An  epoch 
(the  temporal  window  of  analysis)  consisted  of  three  consecutive  overlays.  Fast-Fourier 
transform  (FFT)  was  applied  to  each  overlay  of  the  decontaminated  EEG  signal 
multiplied  by  the  Kaiser  window  (a  =  6.0)  to  compute  the  power  spectral  densities 
(PSDs).  The  PSD  values  were  adjusted  to  take  into  account  zero  values  inserted  for 
artifact-contaminated  data  points.  The  PSD  between  70  and  128  Hz  was  used  to  detect 
EMG  artifacts.  Overlays  with  excessive  EMG  artifacts  or  with  fewer  than  128  data  points 
were  rejected.  The  remaining  overlays  were  then  averaged  to  derive  PSD  for  each  epoch 
with  a  50%  overlapping  window.  Epochs  with  two  or  more  overlays  with  EMG  or 
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missing  data  were  classified  as  invalid.  For  each  channel,  PSD  values  were  derived  for 
each  1-Hz  bin  from  3  to  40  Hz  and  the  total  PSD  from  3  to  40  Hz.  Relative  power 
variables  were  also  computed  for  each  channel  and  bin  using  the  formula  (total  band 
power/total  bin  power). 

2.1.3  Real-Time  Cognitive  State  Classification 

The  use  of  EEG  as  the  basis  for  cognitive  state  assessment  was  motivated  by  charac¬ 
teristics  such  as  good  temporal  resolution,  low  invasiveness,  low  cost,  and  portability. 
While  EEG  offers  several  benefits,  there  were  shortcomings  that  were  addressed  by  this 
research  effort,  including  the  noise  artifacts  described  above  and  the  non-stationarity  of 
neural  signals  over  time  (Popivanov  &  Mineva,  1999).  Despite  these  challenges,  research 
has  shown  that  EEG  activity  can  be  used  to  assess  a  variety  of  cognitive  states  that  affect 
complex  task  performance.  These  include  working  memory,  alertness,  executive  control, 
and  visual  information  processing.  These  findings  point  to  the  potential  for  using  EEG 
measurements  as  the  basis  for  cognitive  state  assessments  in  complex  task  environments 
such  as  dismounted  soldiering. 

Estimates  of  spectral  power  formed  the  input  features  to  a  pattern  classification  system. 
The  classification  system  used  parametric  and  nonparametric  techniques  to  assess  the 
likely  cognitive  state  on  the  basis  of  spectral  features,  i.e.,  to  estimate  p(cognitive  state  \ 
spectral  features).  The  classification  process  relied  on  probability  density  estimates 
derived  from  a  set  of  spectral  samples.  These  spectral  samples  were  gathered  during  the 
same  field  mission  as  the  other  samples  that  would  be  used  to  evaluate  classification 
performance. 

The  classification  system  utilized  a  support  vector  machine  to  discriminate  between  low 
and  high  task  load.  Support  vector  machines  are  linear  classifiers  that  use  a  quadratic 
optimization  procedure  to  find  an  optimal  orientation  and  location  for  a  discriminating 
hyperplane  between  two  classes  of  data.  The  optimization  procedure  finds  a  location  and 
orientation  for  the  hyperplane  that  lies  as  far  away  as  possible  from  examples  in  each 
class  that  were  likely  to  be  confused  with  each  other  (see  Figure  4). 


Figure  4.  Hyperplane  orientation. 
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Separating  hyperplanes  identified  using  this  procedure  has  been  shown  to  maximize 
generalization  performance  (Vapnick,  1999).  Although  they  were  linear  classifiers, 
support  vector  machines  were  used  to  solve  nonlinear  problems  by  means  of  the  so-called 
kernel  trick.  Data  that  may  not  have  been  linearly  separable  in  the  original  feature  space 
were  projected  into  a  high-dimensional  space  where  the  data  may  be  linearly  separable 
(Figure  5).  The  support  vector  machine  used  in  this  effort  employed  a  radial  basis 
function  kernel  with  a  kernel  parameter  of  1  and  a  slack  parameter  of  0.05. 


Figure  5.  Kernel  trick  projection  to  higher  dimensional  space. 

2.1.4  Mobile  Processing  Platform  and  Communications  Network 

Ideally,  to  enable  maximum  mobility,  all  the  processing  power  needed  to  collect  and 
process  the  sensor  data  should  be  located  on  the  body  of  the  soldier.  In  addition  to 
logging  the  data,  the  raw  sensor  data  were  processed  by  the  signal  processing  algorithms 
to  produce  a  clean  signal  for  Honeywell’s  cognitive  state  classification  algorithms. 

That  cognitive  state  assessment  was  transmitted  via  a  communications  network  to  “publish” 
the  information  for  wider  use. 

2.1.5  Commander’s  Display 

Experiments  in  previous  phases  explored  the  feasibility  and  utility  of  “closing  the  loop” 
by  providing  a  company  commander  (CO)  with  real-time  cognitive  state  information  of 
subordinate  platoon  members  (Domeich  et  al.,  2007).  This  was  operationalized  by 
displaying  cognitive  state  information  to  the  CO  to  allow  them  to  adjust  the  flow  of 
communications  to  better  match  the  subordinates’  current  capacity  to  process 
information.  In  previous  evaluations,  Honeywell  explored  using  automation  to  close  the 
loop,  where  the  automation  was  driven  by  assessments  of  cognitive  state  (Domeich  et  al., 
2006;  Domeich  et  al.,  2005).  In  the  2006  evaluation,  the  loop  was  closed  by  a  human 
leader  using  cognitive  state  feedback  of  subordinates  and  then  modifying  the  information 
flow  to  those  subordinates.  This  mitigation  strategy  most  closely  aligned  with  the 
interests  of  the  FFW  ATD,  which  saw  cognitive  state  feedback  as  useful  information  for 
a  leader  when  assessing  the  combat  readiness  of  his  or  her  troops. 
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3  Integration  with  FFW 


3. 1  Overview 

The  primary  goal  of  this  phase  of  the  project  was  to  integrate  the  Honeywell  AugCog 
system  into  FFW  systems.  Specifically,  the  project  worked  toward  full  integration  into 
the  FFW  Early  Increment  2  Leader  System  for  demonstration  in  the  C4ISR  OTM 
technology  demonstration  at  FT  Dix,  New  Jersey,  in  July  2007  (see  Figure  6). 
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Figure  6.  Honeywell  AugCog  system  integrated  into  the  FFW  Network. 


3.1.1  Mobile  Processing  Platform  and  Communications  Network 

The  Early  Increment  2  Leader  System  package  was  a  soldier-worn  hardware  and  software 
ensemble  that  provided  next-generation  digital  capabilities  to  the  dismounted  soldier  (see 
Figure  7).  The  core  of  this  system  was  the  Panasonic  CF-18  Toughbook  laptop  that 
supported  processing  and  the  ITT  Wearable  Soldier  Radio  Terminal  (WSRT)  that 
provided  voice  and  data  communication  to  the  FFW  Network.  The  Honeywell  signal 
processing  and  classification  software  was  hosted  on  the  CF-18,  and  the  output  of  this 
software  was  formatted  as  a  cursor-on-target  (CoT)  message  (standard  information 
schema  that  supported  information  interoperability  and  presentation  of  information  on  all 
FalconView™  displays)  and  transmitted  across  the  FFW  Network  via  WSRT  radios. 
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Figure  7.  The  FFW  Early  Increment  2  Leader  System. 

3.1.2  Commander’s  Display 

Cognitive  state  assessments  of  a  soldier  were  published  over  the  FFW  Network  in  CoT 
format  for  display  on  a  Commander’s  FalconView™  display.  Since  there  is  no  standard 
symbology  definition  for  the  individual  soldier  level,  let  alone  for  cognitive  state 
feedback,  symbology  was  proposed  and  implemented  for  the  OTM  demonstration.  These 
symbols  are  shown  in  Table  1.  See  Appendix  C  for  the  derivation  of  AugCog  symbology 
that  follows  tactical  symbol  conventions  from  the  Common  Warfighting  Symbology  from 
MIL-STD-2525B  (Department  of  Defense,  1999). 


Table  1.  Tactical  symbols  for  cognitive  state  feedback. 
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4  OTM  Experiment 


4.1  OTM  Overview 

In  2007,  the  U.S.  Army  held  the  C4ISR  OTM  technology  demonstration  at  FT  Dix,  New 
Jersey.  The  goal  was  to  showcase  innovative  technologies  that  would  improve  soldiers’ 
effectiveness  by  increasing  their  situation  awareness  in  battlefield  environments  (see 
Figure  8).  The  OTM  involved  FFW  Experimental  Force  (EXFOR)  soldiers  in  realistic 
war  game  scenarios,  utilizing  more  than  100  live  communication,  sensor,  and  battle 
command  systems  (CERDEC,  2007). 


Figure  8.  OTM  Event  at  FT  Dix,  New  Jersey. 


4. 2  Honeywell  Participa tion 

The  Honey well/NSRDEC  team  integrated  AugCog  hardware  and  software  with  FFW 
Leader  Systems  during  the  2007  PM  C4ISR  OTM  event.  The  team  then  collected  ECG 
and  EEG  data  on  the  third  squad  leader  during  force-on- force  exercises  that  were 
conducted  on  July  19,  2007,  at  FT  Dix.  In  addition  to  satisfying  the  integration  goal  of 
this  program,  Honeywell’s  participation  at  OTM  provided  an  additional  opportunity  to 
evaluate  the  ability  to  distinguish  between  low  and  high  cognitive  task  load  in  an 
operational  setting.  The  ability  to  reliably  differentiate  these  two  states  could  be 
leveraged  for  operational  advantage  to  augment  the  commander’s  awareness  of  the  state 
of  his  subordinates,  as  well  as  serving  as  input  to  future  adaptive  systems  that  could  tailor 
information  presentation  and  management  to  the  state  of  the  recipient.  This  section 
documents  the  classification  approach  and  results. 

The  AugCog  system  was  tested  and  data  were  collected  during  two  OTM  exercises  that 
involved  live,  virtual,  and  distributed  assets.  On  the  first  day,  the  platoon  conducted  a 
phased  attack  on  a  MOUT  site,  which  was  defended  by  the  opposing  force  (OPFOR).  The 
second  day  of  data  collection  involved  a  six-hour  mission  where  the  platoon  first 
approached  the  objective  in  High  Mobility  Multipurpose  Wheeled  Vehicles  (HMMWVs), 
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dismounted,  and  moved  to  the  objective  rally  point  (ORP),  which  they  took  by  force.  The 
third  squad  leader  participated  in  a  leader  recon,  where  the  platoon  leader  led  his  squad 
leaders  to  reconnoiter  the  objective.  Rather  than  returning  to  the  ORP,  the  leaders  utilized 
their  FFW  systems  to  relay  the  plan  electronically  to  the  platoon,  which  then  moved  up  to 
take  positions  per  the  plan.  The  third  squad  was  tasked  with  leading  a  feint  to  divert  the 
OPFOR  attention  before  the  main  force  attacked  the  MOUT  site.  The  OPFOR  were 
holding  hostages,  so  there  was  a  mix  of  OPFOR  and  civilians  in  the  battlespace.  The  third 
squad  eventually  attacked  and  occupied  a  building  by  force  before  they  placed  unattended 
ground  sensors  for  remote  monitoring. 

4.3  Equipment  Setup 

The  AugCog  team  collected  EEG  signals  from  ABM’s  wireless  EEG  sensor  headset, 
which  was  integrated  under  the  FFW  helmet  (see  Figure  9);  ECG  data  from  Hidalgo’s 
Vital  Signs  Detection  Unit,  part  of  the  U.S.  Army’s  Warfighter  Physiological  Status 
Monitoring  program,  were  also  collected  and  fused  with  the  EEG  signals  in  order  to 
classify  real-time  cognitive  state  information.  Utilizing  the  FFW  CF-18  Toughbook 
computer  as  a  processing  platform,  signal  processing  algorithms  cleaned  the  sensor 
signals  so  that  advanced  algorithms  could  status  the  third  squad  leader’s  cognitive  state 
on  a  moment-to-moment  basis.  The  resultant  cognitive  state  assessment  information  was 
then  formatted  in  the  CoT  XML  tags  needed  to  push  the  information  over  the  FFW 
network  to  support  the  display  of  AugCog  state  information  on  FalconView  icons  that 
were  viewable  by  those  superior  to  the  sensed  soldier,  platoon  leader,  and  platoon 
sergeant. 


Figure  9.  EEG  setup. 


4.4  Research  Objectives 

The  primary  goal  of  the  AugCog-OTM  experiment  was  to  demonstrate  the  capability  of 
the  AugCog  mobile  system  to  deliver  robust  ECG  and  EEG  signals  and  to  classify 
cognitive  state  classification  in  a  harsh  operational  environment.  The  cognitive  state 
classification  would  output  an  assessment  of  cognitive  workload  into  one  of  two  levels: 
low  workload  and  high  workload.  The  two  levels  of  workload  were  defined  operationally. 
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The  soliders  were  experiencing  low  workload  when  they  had  spare  capacity  to  handle 
task  demands.  In  low  workload,  the  soldier  could  take  on  additional  tasks  without 
compromising  performance  on  current  tasks.  Low  workload  did  not  mean  that  the  soldier 
was  doing  nothing;  rather,  the  soldier  was  handling  the  current  task  load  well  and  could 
take  on  additional  tasks.  Examples  included  mission  lulls  or  the  mission  being  executed 
flawlessly  with  little  variation  from  the  preplanned,  well-versed  drill. 

The  soliders  were  experiencing  high  workload  when  the  task  load  maximized  or  even 
exceeded  the  soldier’s  cognitive  capacity.  The  soldier  absolutely  could  not  take  on  an 
additional  task  without  compromising  performance  on  current  tasks,  and  current  task 
performance  was  possibly  suboptimal  due  to  excessive  cognitive  demands.  Examples  that 
drove  workload  into  the  high  range  included  unexpected  and  unplanned  events  that 
required  rapid  replanning  and  extensive  coordination  by  the  leadership. 

4.5  Classification  Approach 

The  Honeywell  classification  approach  used  EEG  PSD  features,  decomposed  into  the 
different  clinical  bands,  as  input  to  the  classifier.  The  EEG  PSD  features  that  form  the 
basis  for  classification  contain  information  pertinent  to  the  classification  of  cognitive 
states,  as  well  as  irrelevant  components  and  noise.  Accurate  classification  of  workload 
based  on  EEG  called  for  a  system  that  could  estimate  workload  by  identifying 
dimensions  or  features  of  EEG  that  were  informative  with  respect  to  distinctions  among 
workload  levels.  The  analysis  relied  on  the  logistic  classifier.  A  logistic  classifier 
assumes  that  the  relationship  between  a  set  of  independent  variables  (EEG  features  in  this 
context)  and  the  estimated  probability  of  membership  in  a  class  (high  or  low  workload) 
can  be  modeled  in  terms  of  a  sigmoid  function:  P(c|y)  =  l/(l+ey).  Model  parameters  were 
identified  using  maximum  likelihood  estimation.  The  decision  boundary  created  by  this 
classifier  was  linear.  Linear  classifiers  are  widely  used  by  EEG  researchers  as  their 
inherently  low  complexity  limits  the  possibility  of  overfitting  —  an  issue  of  concern  in 
artifact-rich  mobile  task  contexts. 

4.6  Data  Analysis  Methodology 

4.6.1  Ground  Truth 

The  first  step  in  evaluating  a  classification  system  in  the  field  is  to  collect  data  during  the 
exercise  that  can  establish  ground  truth  for  the  phenomenon  of  interest.  In  this  case, 
disparate  task  loads  experienced  by  soldiers  in  the  field,  i.e.,  low  and  high  task  loads, 
were  of  interest.  Accordingly,  a  videographer  was  deployed  to  record  the  activities  of  the 
third  squad  leader  while  he  conducted  a  force-on-force  mission  in  the  pine  barrens  of  FT 
Dix,  New  Jersey.  Next,  multiple  expert  reviewers  analyzed  the  video  footage  to  identify 
periods  of  high  and  low  task  load.  These  reviewers  used  the  following  criteria  to  select 
suitable  periods: 

•  Sustained  periods  (>  5  minutes)  of  consistent  task  load 

•  Periods  that  did  not  contain  transitions  from  low-to-high  or  high-to-low  task  load 

•  Good  EEG  signal  quality  during  the  period 
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It  is  worth  noting  that  the  analysis  was  not  constrained  by  ECG  signal  quality,  since  the 
battery  in  the  Wearable  Physiological  Status  Monitor  (WPSM)  died  prior  to  the  end  of 
the  long  exercise.  As  a  result,  all  classification  results  presented  here  were  based  on  EEG 
data  only. 

4.6.2  Classification  Metric 

A  metric  used  to  evaluate  classification  performance  was  the  area  under  the  Receiver 
Operating  Characteristic  (ROC)  curve  (see  Duda,  Stork,  &  Hart,  2001).  ROC  curves  plot 
true  positives  (on  the  y-axis)  against  false  positives  (on  the  x-axis)  as  a  threshold  for 
discriminating  between  targets  and  distracters.  The  ROC  curve  provides  a  way  to  assess 
the  degree  of  overlap  between  two  univariate  distributions.  It  is  widely  used  to  evaluate 
human  and  machine  signal  detection  capabilities.  The  ROC  curve  provides  a  way  to 
assess  the  degree  of  overlap  between  the  outputs  of  a  classifier  for  two  classes  of  data. 
Perfect  classification  produces  an  area  under  the  curve  value  (Az)  of  1.0,  while  chance 
performance  produces  an  Az  value  of  0.5. 

In  noisy  operational  environments,  EEG  and  other  electrophysiological  sensors  can  be 
compromised  by  noise  over  short  temporal  windows.  One  strategy  for  dealing  with 
momentary  fluctuations  in  classification  accuracy  is  to  median  filter  the  output  of  the 
classifier  over  different  time  windows.  One  consequence  of  such  temporal  smoothing  of 
classifier  output  is  that  it  may  introduce  a  lag  in  the  decision  process.  The  analysis  must 
consider  the  tradeoff  in  accuracy  as  the  temporal  window  of  output  smoothing  is  varied 
(Domeich  et  al.,  in  press).  This  strategy  assumes  that  task  demands  remain  stable  over  the 
span  of  the  smoothing  window.  Smoothing  was  accomplished  using  a  median  filter  on  the 
output  of  the  classifier  over  specific  time  windows.  The  analysis  considered  the  tradeoff 
in  accuracy  as  the  temporal  window  of  output  smoothing  was  varied. 

One  way  to  explore  the  bias  and  variance  of  a  classifier  is  through  a  process  called  n-fold 
cross-validation.  This  procedure  entails  splitting  the  data  into  n  subsets.  At  each  iteration 
of  the  validation  procedure,  one  of  these  subsets  (ni)  was  used  for  testing  the  classifier, 
while  the  remaining  1  -  1/n  sets  were  used  for  training  the  classifier.  A  typical  choice  of 
n  is  10.  Estimates  of  bias  and  variance  get  more  conservative  as  the  size  of  n  decreases — 
the  classifier  has  to  be  trained  with  less  of  the  data  and  is  assessed  by  generalizing  to  a 
larger  subset  of  unseen  data. 

Classification  performance  was  assessed  using  cross-validation.  With  10-fold  cross- 
validation,  the  data  set  was  split  into  10  subsets.  Over  the  course  of  10  iterations,  a  new 
subset  was  picked  to  serve  as  testing  data,  while  the  remaining  9  subsets  served  as  the 
training  data. 

4.7  Results 

4.7 .1  Ground  Truth 

After  expert  review  of  the  video,  two  periods  each  of  low  and  high  task  load  were 
identified,  as  shown  in  Table  2. 
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Table  2.  Periods  of  identifiable  high  and  low  task  load. 


Task 

Load 

Duration 

(mm:ss) 

Description 

High 

05:15 

Squad  leader  was  leading  a  detailed  mission  planning  session  with  his  team 
leaders  immediately  prior  to  attack. 

Low 

08:09 

Squad  leader  was  waiting  for  platoon  leader  to  return. 

Low 

09:45 

Squad  leader  was  sitting  quietly. 

High 

07:57 

Squad  leader  was  positioning  his  men  during  early  phases  of  attack.  He  was 
also  firing  on  enemy  targets  and  coordinating  ongoing  activities. 

In  addition  to  assessing  the  cognitive  task  load  of  each  segment,  pains  were  taken  to 
select  those  segments  of  relatively  low  physical  activity  (e.g.,  standing,  crouching)  and  to 
equate  the  activity  level  between  the  selected  low-  and  high-workload  periods.  Primarily 
this  was  done  to  provide  the  best  quality  EEG  signals  for  the  classification  analysis.  This 
was  also  done  to  eliminate  a  spurious  data  source  (i.e.,  motion  artifacts)  that  could 
inadvertently  drive  the  classification  results,  even  though  it  has  been  previously 
demonstrated  that  the  Honeywell  classification  approach  was  not  driven  by  physical 
activity  (Domeich  et  al.,  2007). 

4.7.2  Classification  Results 

The  results  presented  here  were  from  EEG-based  classification  only  since  the  VSDS’ 
battery  died  prior  to  the  critical  high-workload  periods  near  the  end  of  the  six-hour 
mission.  For  any  classifier  to  work,  there  need  to  be  differences  among  the  features  for 
the  classes  of  interest,  in  this  case,  low  and  high  task  load.  To  highlight  the  inherent 
differences,  the  feature  outputs  were  normalized  and  plotted  against  the  six  bipolar 
channels  (C3C4,  F3Cz,  FzC3,  CzPO,  F3F4,  FzPO)  across  the  clinical  bands  (delta,  theta, 
alpha,  beta,  and  gamma).  Figure  10  illustrates  that  there  was  separation  in  the  average 
power  between  workload  levels  in  all  channels,  and  across  all  bands,  which  formed  the 
basis  of  the  classifier’s  ability  to  distinguish  between  workload  levels. 
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Figure  10.  Feature  amplitude  by  clinical  band  for  all  six  channels  of  EEG. 


Figure  1 1  presents  the  classification  accuracy  as  a  function  of  the  temporal  smoothing 
window. 
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Figure  11.  Classification  accuracy  as  a  function  of  temporal  smoothing. 
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Temporal  smoothing  up  to  one  minute  was  investigated.  Classification  accuracy  rose 
monotonically  up  to  a  one-minute-long  temporal  smoothing  window;  however,  the  rate  at 
which  temporal  smoothing  benefited  accuracy  appeared  to  diminish  as  window  sizes 
increased.  Given  the  criteria  in  workload  period  selection,  it  was  likely  that  task  demands 
remained  stable  up  to  the  one-minute  window. 
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5  Dry  EEG  Electrode  Evaluations 


5.1  Motivation 

The  second  goal  of  this  project  was  to  advance  the  deployability  of  EEG  sensor  systems. 
Toward  that  goal,  Honeywell  integrated  the  QUASAR  RF  dry  EEG  electrodes  under  a 
combat  helmet  in  the  Honeywell  AugCog  system.  This  moved  the  system  that  much 
closer  to  a  deployable  setup.  The  OTM  2007  EEG  data  were  collected  with  a 
conventional  “wet”  electrode  setup  that  required  the  application  of  electrolyte  gel,  which 
has  obvious  soldier  acceptability  and  logistical  issues.  The  QUASAR  dry  electrode 
system  delivered  low  size,  weight,  and  power  performance  and  was  designed  with 
suspension  under  the  helmet  to  maximize  isolation  from  helmet  motion.  This  improved 
signal  quality  and  soldier  comfort.  The  combination  of  NSRDEC’s  domain  expertise  and 
strategic  leadership,  Honeywell’s  signal  processing  and  classification  algorithms,  and 
QUASAR’s  deployable  dry  EEG  sensor  supported  a  compelling  AugCog  offering  with 
superior  soldier  acceptance  and  utility. 

5.2  QUASAR  Dry  Electrodes 

EEG  data  were  collected  from  a  QUASAR’  EEG  sensor  headset  (see  Figure  12).  The 
sensor  headset  acquires  six  channels  of  bipolar  montage  EEG  data  using  eight  sensors. 
Differential  EEG  were  sampled  from  bipolar  channels  CzPOz,  FzPOz,  F3Cz,  F3F4, 
FzC3,  C3C4,  and  a  reference  sensor  at  P4,  all  at  240  samples  per  second  with  a  signal 
bandwidth  from  DC  to  100  Hz.  Data  were  transmitted  across  a  2.4-GHz  wireless  link 
(similar  to  802. 1  lg  in  character)  to  a  base  station  and  then  the  collection  laptop  via  a 
USB  interface.  Quantification  of  the  EEG  data  in  real  time  was  achieved  using  signal 
analysis  techniques  to  identify  and  decontaminate  eye  blinks  and  identify  and  reject  data 
points  contaminated  with  electromyography  (EMG),  amplifier  saturation,  and/or 
excursions  due  to  movement  artifacts. 

The  military  version  of  the  Sensor  Headset  and  wireless  data  acquisition  system 
developed  by  QUASAR  was  made  from  a  modified  FFW  helmet  and  harness.  This  fit 
snugly  on  the  head  and  houses  EEG  sensors  like  many  FDA-approved  laboratory  EEG 
systems.  Unlike  the  conventional  wet  electrode  EEG  sensors,  the  QUASAR  EEG 
electrodes  were  dry  and  required  no  prior  skin  preparation  and  no  conducting  gels  to 
operate.  Physiological  recordings  were  made  with  low-powered  EEG  and  EOG 
amplifiers  and  16-bit  A/D  converters  designed  specifically  for  ambulatory  recordings. 
The  Sensor  Headset  was  only  used  for  recording  physiological  signals  and  did  not 
introduce  energy  into  the  body  except  for  very  minor  electromagnetic  radiation  typically 
emitted  by  small  electronic  devices.  The  only  risk  posed  by  this  device  was  mild 
discomfort  due  to  the  pressure  exerted  by  the  FFW  harness  and  sensors  on  the  user’s 
head.  To  minimize  this  risk,  prior  studies  at  QUASAR  have  assessed  comfort  and 
wearing  time  versus  sensor  load;  the  individual  sensor  loading  was  adjustable  and  was 
measured  for  each  participant  prior  to  testing. 
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Figure  12.  QUASAR’S  wireless  EEG  sensor  headset. 


Two  experiments  were  conducted  to  assess  the  viability  of  the  dry  electrode  technology. 
The  first  was  a  laboratory  experiment,  followed  by  a  field  experiment  where  the  system 
was  used  on  soldiers  performing  operational  tasks  in  a  training  setting. 

5.3  Laboratory  Evaluation 

5.3.1  Research  Objectives 

The  purpose  of  this  experiment  was  to  record  EEG  data  under  controlled  laboratory 
conditions  using  QUASAR’s  hybrid  EEG  sensor  system  (which  features  dry,  wireless 
silver  sensors  exhibiting  capacitive  coupling  at  low  frequencies  and  resistive  coupling  at 
high  frequencies)  to  assess  signal  quality  during  during  the  performance  of  high-  and 
low-workload  tasks  with  helmet  on  and  helmet  off  under  both  static  and  ambulatory 
conditions. 

5.3.2  Method 

5. 3. 2. 1  Experiment  1 :  Auditory  N-back  Lab 

Four  participants  were  tested  in  two  experiments  using  QUASAR’s  hybrid  EEG  sensor 
system.  The  first  experiment  consisted  of  an  n-back  test  under  all  combinations  of  three 
independent  variables  workload  (low,  high),  mobility  (seated,  walking),  and  helmet  (on, 
off):  see  Figure  13.  The  goal  was  to  test  artifact  reduction  methods  and  workload 
classification  algorithms  on  spectral  features  of  the  data. 
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Figure  13.  Independent  variables  in  the  dry  electrode  lab  evaluation. 


The  working  memory  assessment  was  conducted  using  an  auditory  n-back  task.  The 
n-back  task  required  participants  to  process  a  sequence  of  letters  presented  on  a  computer 
screen.  With  every  presentation  of  a  letter,  the  participant  had  to  both  encode  the  letter  in 
memory  and  indicate  whether  the  letter  corresponded  to  a  letter  shown  n  presentations 
ago.  Working  memory  load  encountered  by  a  participant  was  controlled  by  manipulating 
the  value  of  n.  For  this  experiment,  the  low  workload  condition  was  a  0-back  test  (does 
the  current  letter  presented  match  the  first  letter  presented  in  the  entire  sequence?).  High 
workload  was  a  3-back  test  (does  the  current  letter  match  the  letter  from  three 
presentations  ago?).  Participants  responded  affirmatively  by  clicking  the  left  mouse 
button  of  a  wireless  mouse  when  the  current  letter  matched  the  target  (either  the  first 
letter  in  the  sequence  for  0-back  or  the  letter  presented  three  presentations  ago  in  3-back). 
In  the  walking  trials,  participants  walked  around  the  perimeter  of  a  medium-  sized  room 
(24  x  24  feet)  at  a  regular  pace  as  dictated  by  a  metronome  —  to  ensure  that  participants 
walked  at  the  same  pace  for  both  low-  and  high-workload  trials  (since  it  was  suspected 
that  participants  might  inadvertently  slow  down  during  high-workload  blocks  as  more  of 
their  attention  was  focused  on  the  n-back  task).  Walking  pace  was  controlled  to  remove  a 
potential  confound  with  workload  level  and  to  control  the  level  of  motion  artifacts  across 
conditions.  Participants  listened  to  the  letters  played  through  the  PC  speakers  and 
responded  with  a  wireless  mouse  during  the  walking  trials. 

53.2.2  Experiment  2:  Evoked  Response  Potentials  (ERP)  Study 

The  goal  of  the  second  experiment  was  to  test  the  ability  of  the  dry  electrode  system  to 
detect  Evoked  Response  Potentials  (ERP)  associated  with  a  Rapid  Serial  Visual 
Presentation  (RSVP)  task.  The  RSVP  task  consisted  of  rapid  presentations  (every  200 
milliseconds)  of  a  series  of  satellite  images  with  and  without  relevant  missile-launch 
stimuli  (targets).  The  participant  seeing  a  target  provokes  an  observable  ERP  which  can 
be  related  to  well-known  waveform  features  revealed  in  previous  research  (Mathan  et  al., 
2007). 

5.3.23  Data  Analysis 

As  discussed  earlier,  it  was  important  to  know  how  effectively  a  classification  approach 
can  differentiate  between  classes  on  a  moment-to-moment  basis.  The  ROC-curve  metric 
was  used  to  evaluate  classification  performance.  Ten-fold  cross-validation  was  used  to 
evaluate  the  classification  approach. 
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5.3.3  Results 

Overall,  the  sensors  and  hardware/software  integration  between  stimulus  presentation  and 
data  acquisition  systems  performed  well,  and  quality  EEG  data  were  observed,  at  least 
with  preliminary  subjective  analyses,  in  each  of  the  four  participants  during  both 
experiments.  As  with  “gold-standard”  wet  electrode  systems,  more  artifacts  were 
observed  during  dynamic  ambulatory  conditions  than  during  seated  conditions,  and  the 
extent  of  artifact  contamination  varied  by  participant. 

5.3.3. 1  Auditory  n-back  Lab  Results 

Table  3  shows  the  results  from  the  auditory  n-back  lab  classification.  The  classification 
results  were  obtained  using  Ten-fold  cross-validation  and  logistic  regression.  The  area 
under  the  ROC  curve  was  used  as  the  performance  metric.  Participants  1  and  4  did  not 
participate  in  a  helmet-off  condition  due  to  limited  availability. 


Table  3.  Auditory  n-back  lab  classification  results. 


0.45 

0.75 

0.74 

0.90 

0.89 

0.73 

0.70 

0.70 

X 

0.68 

0.82 

X 

0.72 

0.71 

0.74 

0.78 

0.72 

0.69 

0.72 

0.78 

It  is  not  clear  how  to  explain  Participant  4’s  very  low  classification  during  seated  trials. 
The  best  results  for  the  other  three  participants  were  obtained  during  seated  trials,  as 
expected  given  the  better  signal  quality  afforded  by  low  levels  of  mobility.  However,  the 
falloff  in  performance  due  to  motion  varied  depending  on  the  participant  (-0.02  to  ~ 
0.21).  The  results  in  the  “helmet  on”  condition  were  comparable  with  results  seen  with 
wet  electrodes  (Mathan  et  al.,  2007).  The  overall  classification  results  pooled  data  from 
both  mobility  conditions  and  both  helmet  conditions.  Performance  was  in  line  with  what 
has  been  seen  in  previous  laboratory  evaluations. 

5. 3. 3. 2  ERP  Study 

Table  4  lists  the  single  trial  ERP  results  using  QUASAR’s  dry  electrode  system.  These 
results  were  obtained  using  a  nonlinear  support  vector  machine  with  10-fold  cross- 
validation. 

Table  4.  Classification  accuracy  for  ERP  detection. 


pi 

P2 

p3 

p4 

Classification 

Accuracy 

0.94 

0.74 

0.76 

0.97 
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Participants  1  and  4  showed  classification  performance  in  line  with  similar  studies 
conducted  with  32  channels  of  wet  electrodes  (Mathan  et  al.,  2006);  however, 
participants  2  and  3  were  far  below  the  classification  level  expected  for  this  task.  It 
should  be  noted  that  these  dry  electrodes  were  only  prototypes  and  that  the  super-high 
impedance  connection  to  the  participant’s  scalp  was  sometimes  unstable. 

5.3.4  Next  Steps 

Further  analyses  were  conducted  by  both  Honeywell  and  NSRDEC  to  examine  strategies 
for  removing  artifacts  in  dynamic  mobile  conditions  (e.g.,  bandpass  filter  settings, 
independent  component  analysis),  and  different  parameters  for  applying  spectral  analyses 
were  discussed  (e.g.,  window  length,  overlap)  in  terms  of  optimizing  time/ffequency 
resolution  and  bias  of  spectral  estimates.  Honeywell  and  NSRDEC  coordinated  efforts  to 
compare  and  contrast  signal  processing  techniques  that  provide  optimal  data  for 
derivation  of  ERP  (RSVP  task)  and  workload  classification  (n-back  task).  QUASAR 
continued  to  make  improvements  to  the  sensor  system  in  terms  of  comfort  and 
miniaturization,  although  the  overall  consensus  was  that  all  parties  were  confident  that 
the  prototype  system  was  ready  to  be  tested  in  an  operational  environment  on  soldiers 
performing  MOUT. 

5.4  Field  Experiment 

5.4.1  Research  Objectives 

The  Honeywell-QUASAR  AugCog  test  event  was  a  joint  effort  between  the  Aberdeen 
Test  Center  (ATC),  Honeywell,  QUASAR,  and  NSRDEC  to  validate  the  ability  of  dry 
EEG  sensors  to  reliably  detect  cognitive  state  signatures  in  soldiers  executing  high- 
fidelity  training  in  an  operational  environment.  The  AugCog  team  conducted  field  tests  of 
the  QUASAR  dry  electrode  system  in  conjunction  with  a  training  event  for  a  19-man 
element  of  the  7th  Special  Forces  Group  during  the  week  of  March  17,  2008.  The 
objective  of  the  experiment  was  to  test  the  dry  electrode  sensor  system  in  realistic 
conditions,  where  participants  were  stressed  both  physically  and  cognitively. 

5.4.2  Method 

5. 4. 2. 1  Schedule  of  Events 

An  experienced  military  trainer  conducted  the  training  event  as  the  observer/controller 
(O/C).  The  O/C  led  the  troops  in  training  exercises  at  ATC’s  Mulberry  Point  MOUT 
grounds.  The  troops  received  two  days  of  training  prior  to  executing  data  collection  runs 
with  the  EEG  system  in  order  to  familiarize  themselves  with  the  facility  and  reach  a 
certain  level  of  competency  in  the  MOUT  skills. 

On  March  19  and  20,  two  data  runs  were  conducted,  one  in  the  afternoon  and  one  in  the 
evening  (see  Table  5). 
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Table  5.  Participant  schedule. 


Time  of  Day 

March  19 

March  20 

Afternoon 

Team  Leader  1 

Team  Leader  2 

Evening 

Team  Leader  2 

Team  Leader  1 

5. 4. 2. 2  Participants 

Prior  to  the  start  of  the  training  event,  the  element’s  leadership  and  the  O/C  selected  two 
junior  members,  who  would  be  leading  the  training  missions,  to  be  instrumented  with  the 
EEG  system.  The  decision  to  assign  junior  members  to  the  leadership  position  for  the  two 
days  of  data  collection  was  motivated  by  the  need  to  induce  very  high  workload  during 
the  mission,  which  may  have  been  more  difficult  with  senior  leaders. 

5.4.23  Tasks 

For  each  of  the  four  data  collection  runs,  the  assigned  team  leader  (TL)  led  12  soldiers  on 
a  mission  to  “kill  or  capture  an  HVT”  whose  last  known  location  was  in  one  of  the 
buildings  in  Mulberry  Point.  Up  to  six  other  Special  Forces  soldiers  acted  as  OPFOR 
loyal  to  the  high-value  target  (HVT).  The  O/C  and  the  element’s  leadership  managed 
OPFOR  activity  to  achieve  the  experimental  objectives  of  inducing  sustained  periods  of 
high  and  low  cognitive  workload  in  the  instrumented  leader.  All  soldiers  were  armed  with 
simunitions  (i.e.,  soap  bullets)  to  increase  realism  and  stress  during  these  missions. 

Once  the  experimental  team  of  Honeywell  and  QUASAR  had  instrumented  the  soldier 
(see  Figure  14)  and  was  ready  to  record  the  activity  for  “ground  truth”  review,  the 
element’s  leader  briefed  the  participants  on  their  mission  no  more  than  five  minutes  prior 
to  a  required  briefing  of  the  soldiers  they  would  be  leading. 


Figure  14.  Preparing  soldier  participant  for  data  collection. 


The  TL  then  briefed  his  12-soldier  element  on  his  plan  for  achieving  mission  objectives. 
After  a  short  briefing,  the  TL  led  his  soldiers  outside,  where  he  requested  radio  checks 
and  ordered  them  to  load  their  weapons.  The  element  was  then  split  into  two  teams,  and 
each  team  loaded  into  its  HMMWV  for  a  mounted  approach  to  the  objective. 
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5. 4. 2. 4  Data  Collection 

Two  members  of  the  experimental  team  “shadowed”  the  TL  to  record  EEG  that  was 
streamed  wirelessly  from  the  soldier  and  to  videotape  the  activity  to  conduct  post  hoc 
“ground  truth”  assessment  of  low-  and  high-workload  periods.  The  O/Cs  also  followed 
the  movement  of  the  TL  to  support  the  deployment  of  OPFOR. 

The  missions  were  very  fast  paced  and  involved  numerous  engagements  between  OPFOR 
and  Friendly  (Blue)  Force  (BLUFOR).  Radio  communications  were  inconsistent  due  to 
connectivity  issues  exacerbated  by  the  metal  substructures  of  most  of  the  MOUT 
buildings.  This  effectively  induced  high  workload  since  the  TL  had  to  do  more 
replanning  when  he  was  unable  to  communicate  with  the  other  team  under  his  command. 
During  most  missions,  the  TLs  led  their  elements  in  entering  and  clearing  many  buildings 
(6-10)  as  they  moved  toward  their  objectives  and  as  security  required.  The  TLs  also  had 
to  deal  with  wounded  and  killed  soldiers  (casualty  state  determined  by  the  location  of  the 
simunition  hit). 

In  each  case,  the  element  captured  or  killed  the  HVT  before  receiving  authorization  to 
exfiltrate  off  the  objective.  Missions  were  typically  a  frantic  70-90  minutes  of  high  tempo 
activities,  culminating  in  a  hectic  exfiltration  by  vehicle  under  heavy  fire  by  OPFOR. 

Once  BLUFOR  arrived  back  at  the  homebase  building,  the  TL  led  an  after  action  review 
(AAR)  with  the  O/Cs  and  his  element  on  the  mission.  The  element’s  actual  leader 
provided  feedback  on  how  the  mission  was  executed.  The  TL  then  led  a  full  AAR  with 
the  entire  element  to  include  the  perspective  of  OPFOR.  After  the  full  AAR,  the 
experimental  team  removed  the  EEG  system  and  then  reviewed  the  audio/video  footage 
of  the  mission  to  allow  them  to  “talk  through”  the  mission  with  regard  to  their  moment- 
to-moment  mental  workload. 

5.4.3  Results 

A  variety  of  EEG  based  measures  have  been  shown  to  be  sensitive  to  variations  in 
workload  (e.g.,  Wilson  &  Hankins,  1994;  Gevins,  Smith,  McEvoy,  &  Yu,  1997).  EEG 
sensors,  worn  on  the  scalp,  record  electrical  signals  associated  with  neural  activity. 
Signals  recorded  using  these  sensors  were  typically  spectrally  decomposed  for  analysis 
using  the  FFT.  The  FFT  operation  decomposes  each  waveform  into  sinusoidal 
components  that  were  described  by  three  parameters:  amplitude,  frequency,  and  phase. 
The  amplitude  of  EEG  over  various  frequency  bands  -  delta  (1  to  4  Hz),  theta  (4  to  8  Hz), 
alpha  (8  to  13  Hz),  beta  (13  to  30  Hz),  and  gamma  (30  to  40  Hz)  -  have  been  shown  to 
vary  in  conjunction  with  different  brain  states.  For  example,  delta  activity  is  dominant 
during  deep  sleep,  alpha  activity  is  typically  observed  during  wakeful  but  relaxed  states, 
and  beta  and  gamma  activity  is  prominent  during  problem  solving  and  other  complex 
cognitive  tasks  (Scerbo  et.  al.  2001).  While  the  relationship  between  the  EEG  signatures 
mentioned  above  and  various  cognitive  states  were  apparent  in  averages  across  tasks  and 
individuals,  there  was  considerable  variability  in  the  prominence  of  these  features  for  a 
given  individual  in  a  specific  task  context.  In  the  analysis  reported  below,  a  statistical 
machine  learning  process  was  used  to  develop  an  individualized  and  task  specific 
workload  estimator  to  assess  mental  demands  in  each  session. 
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5. 4. 3.1  Classification  Approach 

The  spectral  features  that  form  the  basis  for  classification  contain  both  information 
pertinent  to  the  classification  of  cognitive  states,  and  irrelevant  components/noise. 
Accurate  classification  of  workload  based  on  EEG  calls  for  a  system  that  can  estimate 
workload  by  identifying  dimensions  or  features  of  EEG  that  were  informative  with 
respect  to  distinctions  among  workload  levels.  A  ten-fold  cross-validation  procedure  was 
used  in  conjunction  with  logistic  regression  to  identify  a  subspace  projection  that  would 
permit  discrimination  between  high  and  low  workload  on  the  basis  of  EEG  signals. 


The  linear  projection  described  above  was  optimized  (see  Figure  15)  using  the  logistic 
regression  technique  that  assumes  the  class  conditional  probability  given  the  projection 
followed  a  logistic  model, 


/  =  p(c  I  x)  = 

\  +  ey 


ewT\+b 
l  +  e"’Tx+b 


(1) 


This  likelihood  was  parameterized  by  the  weight  vector  w  and  bias  b.  The  parameters 
were  adjusted  by  maximizing  the  likelihood  of  the  data  so  that  the  data  matches  the 
logistic  model  distribution  in  (1).  The  decision  boundary  created  by  this  classifier  was 
linear.  Linear  classifiers  are  widely  used  by  EEG  researchers,  as  their  inherently  low 
complexity  limits  the  possibility  of  overfitting  -  an  issue  of  concern  in  artifact  rich 
mobile  task  contexts. 


Figure  15.  Example  of  subspace  projection  to  optimize  discrimination  between  classes 

Under  ten- fold  cross-validation,  the  data  were  split  into  ten  subsets.  Over  the  course  of 
ten  iterations,  a  new  subset  was  picked  to  serve  as  testing  data,  while  the  remaining  nine 
folds  served  as  the  training  data.  This  concept  is  illustrated  in  Figure  16,  using  three-fold 
and  two-fold  cross  validation  for  simplicity.  EEG  samples  from  high-  and  low-workload 
conditions  were  used  for  training.  Each  fold  left  out  of  training  was  then  projected  onto  a 
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set  of  weights  identified  using  logistic  regression  in  order  to  derive  a  workload  estimate 
associated  with  the  sample. 


Three  fold  cross  validation 

■■■■■■■■ 

Two  fold  cross  validation 

HHIHH 

Iteration  2 


| training  data 
testing  data 


Figure  16.  N-fold  cross  validation  illustrated  with  n  =3  and  n=2. 

Data  were  split  into  n  equally  sized  segments.  Over  n  iterations  each  data  subset  were 
used  in  turn  for  testing  while  the  remaining  data  were  used  for  training. 

5. 4. 3. 1. 1  Classification  Metric 

The  metric  used  to  evaluate  classification  performance  in  this  effort  was  the  area  under 
the  ROC  curve.  ROC  curves  plot  true  positives  (on  the  y-axis)  against  false  positives  (on 
the  x-axis)  as  a  threshold  for  discriminating  between  targets  and  distracters  was  varied.  It 
is  widely  used  to  evaluate  human  and  machine  signal  detection  capabilities.  The  ROC 
curve  provides  a  way  to  assess  the  degree  of  overlap  between  the  outputs  of  a  classifier 
for  two  classes  of  data.  Perfect  classification  produces  an  area  under  the  curve  value  (Az) 
of  1.0,  while  chance  performance  produces  an  Az  value  of  0.5. 

5. 4.3. 1.2  Qualitative  Analysis 

As  shown  in  the  summaries  of  spectral  features  for  each  participant  in  each  session 
(Figure  17,  Figure  18,  Figure  19,  and  Figure  20),  differences  in  alpha  stood  out  as  the 
most  consistent  pattern  across  participants,  sessions,  and  channels.  Alpha  power  was 
higher  in  the  low-workload  condition.  Differences  in  beta  power  were  also  apparent 
across  workload  conditions.  But,  these  differences  were  less  consistent  across  channels, 
sessions,  and  participants. 
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Figure  17.  Participant  1-  Session  1.  Difference  in  spectral  features  under  low  (green  6+’)  and  high 

(red  V)  workload 
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Figure  18.  Participant  1-  Session  2.  Difference  in  spectral  features  under  low  (green  6+’)  and  high 

(red  V)  workload 
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Figure  19.  Participant  2-  Session  1.  Difference  in  spectral  features  under  low  (green  6+’)  and  high 

(red  V)  workload 
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Figure  20.  Participant  2-  Session  2.  Difference  in  spectral  features  under  low  (green  6+’)  and  high 

(red  V)  workload 
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5. 4. 3. 1. 3  Classification  A  nalysis 

These  classification  results,  as  summarized  in  Figure  21,  point  to  the  feasibility  of 
accurate  cognitive  state  estimation  in  challenging  field  settings.  Average  base 
classification  on  a  sample-by-sample  basis  was  0.75. 
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Figure  21.  Classification  accuracy  as  a  function  of  smoothing  (for  both  participants  in  both  sessions) 

Temporal  smoothing  was  considered  as  a  strategy  for  dealing  with  intermittent 
classification  errors  stemming  from  the  noise  inherent  in  the  field  environment.  This 
strategy  assumes  that  task  demands  remain  stable  over  the  span  of  the  smoothing 
window.  Smoothing  was  accomplished  using  a  median  filter  on  the  output  of  the 
classifier  over  specific  time  windows.  One  consequence  of  temporal  smoothing  of 
classifier  output  is  to  introduce  a  lag  in  the  decision  process.  The  analysis  considered  the 
trade  off  in  accuracy  as  the  temporal  window  of  output  smoothing  was  varied. 

Classification  accuracy  for  both  participants  rose  monotonically  up  to  a  one-minute-long 
temporal  smoothing  window.  However,  the  rate  at  which  temporal  smoothing  benefits 
accuracy  appeared  to  diminish  as  window  size  increased.  Temporal  smoothing  of  ten 
seconds  contributed  to  a  rise  in  classification  accuracy —  average  accuracy  rose  from 
0.75  to  0.87.  With  the  exception  of  Session  1  for  participant  2  -  classification  accuracy 
reached  .90  or  higher  for  both  participants  with  10  seconds  of  temporal  smoothing, 
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6  Conclusions  and  Next  Steps 


The  success  at  FT  Dix  represents  an  advance  in  integration  maturity  for  AugCog  since 
the  system  was  tested  at  the  ATC  in  2006.  During  that  test,  the  EEG  and  ECG  signals 
were  transmitted  wirelessly  to  an  off-the-body  system.  During  the  OTM  2007,  the 
sensing  hardware  and  signal  processing  software  were  hosted  on  the  solider-bome  FFW 
Leader  System.  Likewise,  fielding  dry  EEG  electrodes,  which  supported  comparable 
classification  performance  to  wet  electrodes,  was  another  step  toward  improving  the 
deployability  of  a  sensor-based  cognitive  state  classification  system  such  as  the 
Floneywell  AugCog  systemdescribed  in  this  report. 

To  date,  the  constraints  of  field  test  execution  have  precluded  sufficient  data  collection  to 
develop  robust  models  of  classification  that  would  perform  well  across  conditions  and 
timeframes.  Additional  research  is  required  to  validate  the  extent  to  which  classification 
modelscan  generalize.  Potential  follow-up  studies  should  require  collaboration  with  a 
U.S.  Armypartner  experienced  in  manipulating  tonic  states  during  long-term  studies.  It 
would  be  important  to  conduct  long-term  data  collection  in  an  operational  environment 
with  “ground  truth”  for  task  load,  fatigue,  stress,  sleep  deprivation,  and  physical  load  in 
order  to  develop  larger  data  sets  to  support  improved  model  development  and  more 
robust  classifier  performance.  The  goal  would  be  to  enable  real-time  classification 
approaching  90%  with  historical  participant  models.  In  addition,  future  studies  could 
investigate  classification  performance  as  a  function  of  tonic  states  and  workload  across 
multiple  time  frames.  The  feasibility  of  building  a  general  classifier  that  works  across 
tonic  state  levels  should  be  investigated  to  determine  whether  it  is  necessary  to  build 
separate  classifiers  for  each  tonic  state  level.  Finally,  the  fusion  of  slower-changing  tonic 
state  information  (arousal,  stress,  fatigue,  etc.)  with  faster  changing  moment-to-moment 
oscillatory  EEG  might  yield  more  discriminating  models. 
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Appendix  A 
List  of  Acronyms 


Acronym 

Description 

AAR 

After  Action  Review 

ABM 

Advanced  Brain  Monitoring,  Inc. 

A/D 

Analog-to-digital  (converter) 

ARL 

Army  Research  Lab 

ATC 

Aberdeen  Test  Center 

ATD 

Advanced  Technology  Demonstration 

AugCog 

Augmented  Cognition 

BLIT 

Battle  Lab  Integration  Team 

BLUFOR 

Friendly  (Blue)  Force 

C4ISR 

Command,  Control,  Communications,  Computers,  Intelligence, 
Surveillance,  and  Reconnaissance 

CLIP 

Closed-Loop  Integrated  Prototype 

CO 

Company  Commander 

CONOPS 

Concept  of  Operations 

CoT 

Cursor  on  Target 

CP 

Command  Post 

DARPA 

Defense  Advanced  Research  Projects  Agency 

ECG 

Electrocardiogram 

EEG 

Electroencephalogram 

EMG 

Electromyogram 

EOG 

Electro-oculogram 

ERP 

Evoked  Response  Potential 

EXFOR 

Experimental  Force 

FCS 

Future  Combat  Systems 

FDA 

Food  and  Drug  Administration 

FFT 

Fast-Fourier  Transform 

FFW 

Future  Force  Warrior 

FRAGO 

Fragmentary  Orders 

FT 

Fire  Teams 

HMMWV 

High  Mobility  Multipurpose  Wheeled  Vehicle 

HRED 

Human  Research  and  Engineering  Directorate 

HYT 

High-Value  Target 
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IBI 

Interbeat  Interval 

I/O 

Input/Output 

MOUT 

Military  Operations  in  Urban  Terrain 

NCO 

Network-centric  Operations 

NCW 

Network-centric  warfare 

NSRDEC 

Natick  Soldier  Research,  Development  and  Engineering  Center 

O/C 

Observer/Controller 

OPFOR 

Opposing  Force 

ORP 

Objective  Rally  Point 

OTM 

On-the-Move 

PDA 

Personal  Digital  Assistant 

PL 

Platoon  Leader 

PSD 

Power  Spectral  Densities 

QUASAR 

Quantum  Applied  Science  and  Research 

recon 

Reconnaissance 

RF 

Radio  Frequency 

ROC 

Receiver  Operating  Characteristic 

RSVP 

Rapid  Serial  Visual  Presentation 

RTO 

Radio-Telephone  Operators 

SCU 

Small  Combat  Unit 

TL 

Team  Leader 

up 

Tactics,  Techniques,  and  Procedures 

UI 

User  Interface 

VSDS 

Vital  Signs  Detection  System 

WPSM 

Wearable  Physiological  Status  Monitor  (system) 

WSRT 

Wearable  Soldier  Radio  Terminal 
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Appendix  B 

Augmented  Cognition  System  Small  Unit  CONOPS 


B.  1  General  Introduction 

“Network-centric  warfare  (NCW),  now  commonly  called  network-centric  operations 
(NCO),  is  an  emerging  theory  of  war  in  the  information  age  that  seeks  to  translate  an 
information  advantage  into  a  competitive  warfighting  advantage  through  the  robust 
networking  of  well-informed  geographically  dispersed  forces  allowing  new  forms  of 
organizational  behavior.  This  “networking”  utilizes  information  technology  via  a  robust 
network  to  allow  increased  information  sharing,  collaboration,  and  shared  situational 
awareness,  which  theoretically  allows  greater  self-synchronization,  speed  of  command, 
and  mission  effectiveness.”  (Wikipedia,  2007) 

The  proposed  system  has  four  basic  tenets: 

a.  A  networked  force  improves  information  sharing; 

b.  Information  sharing  enhances  quality  of  information  and  team  situation 
awareness; 

c.  Team  situation  awareness  enables  collaboration  and  synchronization  and 
speed  of  command;  and 

d.  These,  in  turn,  dramatically  increase  mission  effectiveness. 

The  recently  completed  Future  Force  Warrior  (FFW)  Advanced  Technical  Demonstration 
(ATD)  focused  on  NCO  technology  wherein  members  of  a  platoon  size  unit  are 
interconnected  via  electronic  communications.  Work  on  the  future  small  combat  unit 
(SCU)  maintains  this  same  focus  on  communication  technologies.  With  the  advance  of 
this  technology,  modem  infantry  platoons  are  evolving  from  tight  formations  with  one  or 
two  radio-telephone  operators  (RTO)  with  legacy  radio  technology  to  disbursed  teams 
where  every  soldier  is  equipped  with  some  form  of  advanced  digital  communication 
device  (e.g.,  radio-phone  with  display,  PC  tablet,  Personal  Digital  Assistant  (PDA)) 
(Micro  Analysis  &  Design,  2004;  Infantry  Magazine,  2006).  Tactically,  this  allows  the 
platoon  to  spread  out  in  urban  environments  where  a  platoon  leader  (PL)  can  maintain 
contact  with  his/her  squad  and  team  leaders  for  tactical  control  even  under  conditions  of 
low  visibility  (fog,  sand  storms,  night  ops)  (Future  Force  Warrior  Small  Unit  CONOPS, 
2005).  The  potential  down  side  is  that  the  PL  will  lose  some  immediate  awareness  of  the 
soldiers’  physical  and  mental  states  normally  gained  from  direct  observation.  Because 
effective  decision  making  about  which  squad  or  fire  team  member  should  execute  a 
tactical  movement  is  often  made  by  judging  the  physical  and  mental  readiness  of  one’s 
soldiers,  there  will  be  a  need  to  regain  information  lost  in  the  new  technologically  based 
formations  (Akavia  &  Gofer,  2006). 

The  AugCog  system*,  which  originated  as  a  Defense  Advanced  Research  Projects 
Agency  (DARPA)  program,  is  designed  to  provide  SCU  leaders  with  real-time 


*  Co-joined  with  the  Wearable  Physiological  Status  Monitor  (WPSM)  system. 
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information  on  the  psychophysical  readiness  of  their  soldiers.  This  is  achieved  through 
the  use  of  body- worn  sensors  and  microprocessor  classification  algorithms.  The  derived 
cognitive  state  information  can  then  be  fed  to  an  intelligent  information  system  directly  to 
the  leadership  via  the  net-centric  technology.  Leaders  will  have  either  head-mounted 
displays  or  hand-held  devices  that  can  present  graphical  summaries  of  individual  and 
team  states  of  readiness.  The  PL  will  be  able  to  maintain  team  situation  awareness  and 
thus  control  teams  in  broad  formations  and  difficult  terrain  or  conditions. 

The  sensors  worn  by  soldiers  are  expected  to  be  part  of  their  uniform/equipment  and  will 
not  require  any  special  attention  or  care  by  the  soldier.  Thus,  the  system  is  to  be  a 
seamless  component  of  an  overarching  warfighting  system  of  systems  that  serves  the 
solider  and  leader  and  is  value  added  versus  mission  demanding. 

The  AugCog  system  will  not  only  provide  current  cognitive  state  information,  but 
information  about  the  recent  “history”  of  each  soldier’s  cognitive  state.  For  example,  a 
soldier’s  cognitive  load  may  currently  be  low,  but  recent  history  shows  an  exceedingly 
high  workload  for  a  prolonged  period,  in  which  case  that  soldier  may  be  less  ready  than 
one  who  has  a  current  high  workload  but  a  recent  history  of  low  demand.  Therefore,  the 
AugCog  system  will  indicate  cognitive  readiness  states  of  individual  soldiers.  In  addition, 
cognitive  state  feedback  can  be  aggregated  to  display  “Team-at-a- glance”  information 
that  allows  leaders  to  assess  a  team’s  combat  effectiveness  and  readiness  from  a  cognitive 
load  perspective. 

B.2  Assumptions 

B.2.1  Assumptions  Of  CONOPS 

The  AugCog  CONOPS  adopts  the  same  purpose  and  assumptions  as  established  by  the 
FFW  system,  as  expressed  in  the  following  quote: 

“CONOPS  are  normally  developed  when  there  is  either  a  change  in  basic  organizational 
structure  of  a  unit ...  or  when  new  technologies  and  equipment  are  overlaid  on  an  existing 
organizational  structure.  This  is  the  case  with  FFW.  The  program  is  not  creating  a  new 
organizational  structure,  rather  FFW  and  its  embedded  technologies  are  being  overlaid  on 
both  Stryker  (Current  Force)  and  FCS  [Future  Combat  Systems]  (Future  Force)  Infantry 
small  units.  For  purposes  of  this  document,  a  small  unit  is  defined  as  Infantry  Platoon  and 
below.  Although  there  are  various  uses  and  users  of  CONOPS,  the  primary  purpose  of  the 
FFW  CONOPS  is  to  provide  the  vision  for  how  systems/capabilities  are  operated  and 
utilized  at  the  small  unit  level.”  (Future  Force  Warrior  Small  Unit  CONOPS,  2005) 

B.2.2  Assumptions  of  the  AugCog  System 

Given  the  expected  change  of  the  operational  mode  of  the  FFW  SCU  leaders  (viz., 
operating  more  remotely  from  subordinate  units),  the  AugCog  system  provides 
information  about  unit  status  usually  gained  by  direct  visible  contact.  In  addition,  the 
system  is  expected  to  provide  information  about  the  psychophysical  states  of 
subordinates  that  are  not  normally  observable.  The  AugCog  gauges  will  provide  internal 
state  information  about  cognitive  and  physical  readiness  of  key  individuals  that  are  not 
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always  displayed  in  outward  behaviors.  And  last,  the  AugCog  system  will  be  able  to  feed 
directly  into  other  technologies  for  subsystem  automation. 

B.3  Mission  For  Net-Centric  SCUs 

NCW  changes  the  way  commanders  look  at  their  armies.  Instead  of  a  contest  of  numbers 
(my  3,000  troops  can  beat  your  1,000  troops),  the  U.S.  Army  becomes  one  entity  with 
many  parts  that  can  shift  and  adapt  to  quickly  developing  situations.  Information  is 
shared  across  the  entire  network. 

According  to  a  press  release,  “The  Army  transformation  requirements  includes  the  ability 
to  put  a  combat-capable  brigade  anywhere  in  the  world  within  96  hours,  a  full  division  in 
120  hours,  and  five  divisions  on  the  ground  within  30  days.  One  way  to  increase  strategic 
agility  is  to  allow  fewer  soldiers  to  do  more  work  (emphasis  added).”  (Grabianowski, 
2007) 

Thus,  the  concept  of  “fewer”  may  be  mirrored  down  to  companies  and  platoons  or  even 
lower.  Companies  may  be  sent  on  missions  expected  of  battalions,  and  platoons  may  be 
expected  to  do  company-sized  operations.  Information  net-centric  technology  (the  larger 
support  system  of  small  units)  is  expected  to  enable  this  capability.  Relevant  here  is  a 
statement  from  the  2005  FFW  CONOPS  about  SCU  missions  (Future  Force  Warrior 
Small  Unit  CONOPS,  2005). 

B.4  Operations:  An  Illustration 

Given  an  assault  and  secure  mission  of  an  urban  objective,  the  tasks  include  isolating  the 
objective,  gaining  a  foothold  on  the  objective,  systematically  clearing  rooms  and 
structures,  and  finally  securing  and  defending  the  isolated  objective.  The  action  begins 
with  Fragmentary  Orders  (FRAGO)  from  the  company  commander  (CO).  However, 
rather  than  going  to  the  company  command  post  (CP),  the  PLs  and  staff  will  “meet”  with 
the  CO  at  the  platoon  site  via  Net-Comms.  In  the  virtual  CP,  the  PL  obtain  mission 
orders.  PLs  may  interact  with  the  CO  while  the  squad  leaders  (SL)  and  fire  team  leaders 
(TLs)  observe  via  their  display  devices. 

The  PLs  then  meet  with  their  subordinate  leaders,  initially  face-to-face;  then,  as  they 
move  to  the  objective  rally  point  (ORP),  they  will  interact  increasingly  via  their  displays 
and  local  networks,  depending  on  unit  formations  and  conditions.  During  mission 
execution,  plans  are  refined  as  information  is  acquired  from  intelligence  feeds  from 
remote  sensors,  forward  observers,  and  scouts.  PLs  and  SLs  develop  and  lay  out  specific 
plans  of  attack  for  their  sector.  SLs  then  meet  face-to-face  with  their  fire  teams  (FTs)  and 
establish  FT  responsibilities.  Leaders  and  fire  team  members  select  and  prepare  their 
equipment.  Reconnaissance  (recon)  teams  may  still  do  “eyes-on”  reconnaissance,  but  the 
PL  would  also  get  additional  intelligence  through  remote  sensors  and  the  network.  The 
assault  may  be  executed  as  planned  in  a  manner  not  too  different  from  current  tactics, 
techniques,  and  procedures  (TTP),  but  as  action  and  events  unfold,  decisions  about  who 
does  what  must  be  revised.  Whereas  previously  PLs  were  co-located  with  many  of  their 
subordinates,  in  future  operations  where  fewer  soldiers  conduct  larger  scale  operations, 
the  PL  will  be  coordinating  the  mission  largely  through  his  or  her  communication  device. 
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Via  the  AugCog  systems’  cognitive  state  feedback,  the  PL  is  able  to  maintain  situation 
awareness  of  subordinates’  mission  effectiveness  and  current  workload.  As  the  PLs  focus 
on  display  mission  maps,  the  AugCog  summary  icons  will  help  them  gauge  the  states  of 
the  squads  and  fire  teams. 

B.5  AugCog  System  Characteristics 

B.5.1  General  Description 

The  system  comprises  a  variety  of  sensors,  which  can  be  used  individually  or  together  as 
a  sensor  suite.  The  sensor  suite  consists  of  wearable  (body-  and  head-borne),  unobtrusive 
mobile  computer  systems  capable  of  delivering  clean  electrocardiogram  (ECG)  and 
electroencephalogram  (EEG)  signals  in  harsh  operational  environments.  ECG  signals  can 
be  obtained  from  the  Wearable  Physiological  Status  Monitor  (WPSM)  system  that  would 
not  require  any  additional  sensing  equipment.  Adding  EEG  sensors  will,  however,  require 
modification  of  the  helmet  to  add  dry  electrodes  capable  of  detecting  EEG  signals.  The 
existing  mobile  computing  platform  will  host  advanced  signal  processing  capability  to 
improve  sensor  measurements.  For  instance,  signal  processing  to  improve  heart  rate 
(HR),  HR  variability,  and  interbeat-interval  (IBI)  signals  can  be  used  to  improve  the 
WPSM  system  and  casualty  care  metrics. 

The  system  provides  real-time  cognitive  state  classification  processing  from  either  ECG, 
EEG,  or  fused  ECG  and  EEG  signal  components.  The  soldier  is  therefore  outfitted  with  a 
mobile  sensor-based  ensemble  that  monitors  states  of  cognitive  attention  and  readiness 
during  tactical  movement.  The  system: 

•  Integrates  sensor-driven  classification  of  cognitive  state  to  detect  a  change  in  the 
soldier’s  cognitive  state  between  low-task- load  and  high-task- load  conditions, 

•  Provides  information  about  soldier  cognitive  states  to  leadership  displays  that 
reflect  potential  changes  in  operational  performance, 

•  Uses  the  ECG-derived  HR  variability  signal  to  reliably  distinguish  between 
different  cognitive  workload  states, 

•  Can  additionally  use  a  differential  EEG  system  to  provide  the  EEG  data  required 
to  distinguish  cognitive  state  levels, 

•  Can  use  additional  sensors  (e.g.,  accelerometers)  to  improve  cognitive  state 
classification  by  providing  context  information  (i.e.,  accelerometers  to  distinguish 
running  from  stationary)  for  understanding  the  cognitive  state  feedback. 

B.5.2  Description  of  User  Interface  (UI)  Display  and  Input/Output  (I/O)  Devices 

Near-term  application  of  cognitive  state  information  utilizes  a  leader’s  ability  to  “close 
the  loop”  by  providing  PLs  and  a  CO  with  real-time  cognitive  state  information  of 
subordinate  platoon  members.  Displaying  cognitive  state  information  to  leaders  allows 
them  to  adjust  the  flow  of  communications  to  better  match  the  subordinate’s  current 
capacity  to  process  information.  This  mitigation  strategy  most  closely  aligns  with  the 
interests  of  the  FFW  ATD,  which  sees  cognitive  state  feedback  as  useful  information  for 
a  leader  when  assessing  the  combat  readiness  of  his/her  troops. 
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Cognitive  state  information  of  the  subordinates  is  displayed  to  the  PL  or  CO  via  the 
commander’s  display.  For  instance,  a  CO’s  commander’s  display  would  relay 
information  pertaining  to  the  cognitive  state  of  the  PL  and  the  platoon  sergeant.  The  CO’s 
display  might  show  the  current  real-time  assessment  of  cognitive  state  via  a  color-coded 
text  box,  where  the  capacity  of  the  soldier  relative  to  the  task  demands  is  labeled 
“Unknown”  (blue),  “Spare  Capacity”  (green),  “At  Capacity”  (yellow),  or  “Exceeds 
Capacity”  (red).  In  addition,  the  history  of  the  moment-to-moment  assessment  of  a 
soldier’s  cognitive  state  is  shown  via  a  line  graph.  The  background  could  be  redundantly 
color  coded  to  support  “at  a  glance”  processing.  The  scale  of  the  timeline  should  be  user- 
controllable. 

In  addition,  aggregate  measures  of  a  team’s  current  workload  could  be  displayed  to 
identify  possible  mismatches  between  expected  workload  and  the  actual  workload.  This 
would  allow  a  leader  to  maintain  situation  awareness  of  the  current  combat  readiness  of 
groups  of  soldiers  at  a  glance. 

B.6  Aug  Cog  as  an  SCU  Enabler 

One  of  the  core  capabilities  of  the  Army  Transformation  is  the  unparalleled  connectivity 
via  netted  communications,  enabling  information  sharing  (Parmentola,  2004).  Real-time 
collaboration  enhances  the  kind  of  situational  understanding  that  drives  decisive  actions. 
The  inundation  of  information  can  be  expected  to  grow  between  soldier  and  ground  and 
air  sources.  The  potential  data  overload,  coupled  with  the  efficiency  of  information  flow 
required  in  executing  Army  doctrine,  places  an  overabundance  of  critical  information 
throughput  on  a  single  point  of  contact,  the  individual  soldier.  A  means  to  help  manage 
information  overload  of  an  individual  soldier’s  mental  and  cognitive  state  is  needed 
beyond  that  provided  by  external  means  (Dorneich  et  ah,  2006). 

Often  it  is  the  PL  or  the  CO  that  controls  the  flow  of  information  to  subordinates  as  they 
manage  an  operation.  The  situation  changes  rapidly,  requiring  rapid  replanning  and 
continual  updating  of  the  current  status  of  the  soldiers,  current  locations,  and  enemy 
movements  and  intentions.  The  PL  must  understand  the  current  capabilities  of  the 
platoon’s  soldiers  in  order  to  best  utilize  them  to  drive  the  action  forward.  Understanding 
the  individuals’  or  team’s  current  workload  provides  crucial  information  to  the  PL  to 
rapidly  deploy  teams  to  tasks  that  are  best  suited  to  their  current  demands.  In  addition, 
knowledge  of  the  individuals’  or  team’s  current  workload  allows  the  PL  to  adapt  the 
information  flow  to  the  current  capacity  of  the  soldier  to  receive  and  understand  that 
information. 

Some  training  will  be  required  to  help  leaders  understand  how  to  interpret  and  utilize 
cognitive  workload  feedback  appropriately.  As  experience  grows,  it  will  become  clear 
when  cognitive  state  feedback  is  most  useful. 
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B.7  Mission  Operations 

B.7.1  Reconnaissance 

When  a  PL  sets  up  a  leader  recon,  he/she  takes  the  risk  of  having  some  of  his/her  most 
essential  platoon  members  clustered  together,  creating  the  potential  for  serious  loss  as 
they  attempt  to  collect  information  about  the  target  and  the  terrain.  If  there  could  be  more 
separation  among  the  recon  team  members,  the  risk  of  major  team  loss  would  be  reduced. 
However,  separation  from  the  team  reduces  the  PL’s  awareness  of  his  soldiers’ 
conditions  and  states.  Remote  sensors  for  understanding  the  cognitive  states  and 
conditions  of  the  soldiers  would  facilitate  decisions  about  how  to  move  and  direct  the 
team.  Workload  gauges  might  also  indicate  difficult  points  along  the  approach.  This  all 
could  be  done  with  a  smaller  team.  In  fact,  with  sufficient  sensing  and  communication, 
the  PL  could  remain  back  at  the  ORP  and  direct  a  small  recon. 

If  the  PL  is  directing  a  leader  recon  remotely,  he  or  she  would  use  a  larger  display 
platform  like  a  tablet  PC.  Given  the  smaller  number  of  soldiers  involved  in  this  mission, 
he  or  she  would  have  greater  detail  and  more  sensor  information  on  individual  team 
members  and  conditions  than  with  a  larger  team  (see  offensive  operations). 

B.7.2  Offensive  Operations 

When  a  PL  sets  up  offensive  operations,  he  or  she  takes  the  same  risks  as  for  the  leader 
recon.  Again,  if  there  could  be  more  separation  among  team  members,  the  risk  would  be 
reduced.  Here,  too,  remote  sensors  for  understanding  cognitive  states  and  conditions  of 
key  soldiers  would  facilitate  decisions  about  how  to  move  and  direct  the  squads,  and  the 
operations  could  be  done  remotely.  Workload  gauges  would  indicate  difficult  paths  of 
movement  and,  with  other  technologies  for  navigation,  would  help  in  decision  making. 

Unlike  for  the  leader  recon,  the  display  would  have  less  information  on  many  of  the 
individual  team  members.  In  fact,  for  multi-squad  operations,  the  PL  mainly  needs  the  SL 
gauge  and  perhaps  a  summary  gauge  for  the  fire  teams.  A  combination  of  the  SL  and  the 
subordinate  cluster  information,  along  with  radio  communications,  may  be  sufficient  for 
the  PL  to  “read  and  understand”  the  cognitive  states  of  his  squads.  The  more  experience  a 
PL  has  with  this  combined  information,  the  better  he/she  would  be  able  to  build 
situational  knowledge  for  offensive  operations. 

B.7.3  Defensive  Operations 

Defensive  operations  may  not  need  the  same  level  of  remote  sensing  as  either  leader 
recon  or  offensive  operations  except  on  active  patrols.  In  the  case  of  defensive  patrols, 
the  remote  sensing  technology  would  work  the  same  as  in  offensive  operations. 

In  the  case  of  static  defenses,  where  there  is  little  or  no  action,  soldiers  may  become 
inattentive  and  may  suffer  decrements  in  cognitive  vigilance.  When  the  leadership  is 
aware  of  this  lowered  state  among  soldiers,  it  interacts  with  them  and  request  actions  that 
will  improve  their  states  of  alertness.  This  function  can  also  be  automated  so  as  not  to 
require  action  by  the  leaders. 
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B.8  Training 

B.8.1  Training  Requirements  of  System 

The  AugCog  technology  may  not  be  totally  intuitive  to  an  untrained  operator.  The 
technology’s  advantage  may  only  be  realized  after  leaders  have  gained  experience  using 
it  and  have  developed  the  skills  to  more  accurately  “read”  the  states  of  their  men  under 
the  context  and  conditions  of  the  operation.  This  is  not  unlike  the  situation  for  radar 
operators.  Once  the  skill  of  using  the  technology  has  been  acquired,  the  advantage  of 
using  it  versus  not  using  it  will  be  more  fully  appreciated.  Hence  extended  training  will 
be  required  on  this  and  other  sensor-display  technologies.  Interestingly,  the  technology 
itself  can  be  used  to  monitor  and  accelerate  the  training  process.  The  workload  gauges 
can  be  used  to  indicate  where  extra  training  may  be  required.  The  gauges  may  also 
indicate  where  changes  might  be  needed  in  operational  training.  Thus,  the  technology 
will  perform  multiple  roles. 

B.8.2  Training  Applications 

The  AugCog  system  can  be  used  during  training  to  determine  the  impact  of  training 
modules  or  routines  on  the  development  of  skills.  As  certain  skills  improve,  the  gauges 
will  show  reductions  in  cognitive  workload  during  training  of  specific  tactical  operations. 
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Appendix  C 

FFW-OTM  Augmented  Cognition  Symbology 


C.  1  Objective 

Definition  of  appropriate  symbology  to  represent  individual  soldiers  and  appropriate 
modifiers  to  display  cognitive  state  assessment  derived  from  AugCog  soldier-worn 
systems. 

C.2  References 

[FFT  MSIT]  FFW  Medical  Systems  Integration  Team:  FFW  -  OTM  ’07  Exercise: 
Remote  Personnel  Monitoring  Symbology  (vl.2),  9  April  2007. 

[2525B]  Department  of  Defense  Interface  Standard:  Common  Warfighting 
Symbology  (MIL-STD-2525B  w/CHANGE  1,  1  July  2005). 

C.3  Definitions 

•  Electroencephalogram  (EEG):  neurophysiologic  measurement  of  the  electrical 
activity  of  the  brain  by  recording  from  electrodes  placed  on  the  scalp. 

•  Electrocardiogram  (ECG):  physiological  measurements  of  the  electrical  activity 
of  the  heart  over  time. 

•  Cognitive  State  Classification  (CSC):  real-time  determination  of  a  soldier’s 
current  cognitive  workload  from  sensor-based  (EEG  and  ECG)  signals. 

•  Augmented  Cognition:  real-time,  sensor-based  cognitive  state  classification 
systems. 

C.4  Strategy 

Since  a  standard  symbology  definition  for  the  individual  soldier  level  does  not  exist,  the 
proposed  symbology  for  cognitive  state  has  followed  the  lead  of  the  Futire  Force  Warrior 
(FFW)  Medical  Systems  Integration  Team  (MSIT)  approach.  Since  this  document  is 
modeled  after  the  MIST  draft,  the  symbology  proposed  here  is  modeled  after  that 
proposed  by  the  MSIT  (see  [FFT  MSIT]). 

C.5  Symbology 

C.5.1  Soldier  Symbol 

As  of  this  report,  there  is  no  standard  2525B  tactical  symbol  definition  for  the  individual 
soldier.  When  a  2525B  symbol  is  defined  and  is  used  for  specific  soldiers,  the  2525B 
definition  will  take  precedence.  But  for  the  FFW  OTM  2007,  the  new  generic  soldier 
symbology  proposed  by  the  FFW  MSIT  was  used  (Figure  C-l). 
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C\ 

New  generic  soldier  symbology  l 

Reference:  [FFT  MSIT],  page  1. 

Figure  C-l.  New  generic  soldier  symbology. 


The  features  of  the  symbol  (listed  in  Table  C-l)  will  follow  the  MIL-STD-2525B 
specification  for  the  composition  of  tactical  symbols  (2525B,  section  5.4,  page  46).  See 
this  specification  for  definitions  of  the  form  and  fill  of  the  proposed  individual  soldier. 

Table  C-l.  Feature  specifications  for  the  soldier  symbol. 


Feature 

Specification 

Reference 

Frame 

Circle  for  the  individual  soldier. 

[FFW  MSIT] 

Affiliation 

Friend  (only).  Since  there  is  no  cognitive 
state  classification  information  on  non¬ 
friendly  forces,  “Friend”  is  the  only 
affiliation. 

[2525B],  Section  5.3.1 .1 ,  page  48 

Fill 

Crystal  Blue  (128,224,255)  for  friendly 
affiliation. 

[2525B],  Table  XIII,  page  51 

Icon 

Person  icon  illustrated  in  Figure  C-1 

[FFW  MSIT],  page  1. 

ID 

“...the  tactical  ID  for  each  soldier  needs  to 
be  appended  to  his  individual  symbol  to 
ensure  that  he  can  be  properly  identified 
on  the  Common  Operating  Picture.  Since 
this  is  a  tactical  requirement  rather  than 
medical  specific,  that  tactical  ID  will  be 
appended  to  this  symbol.” 

[FFW  MSIT],  page  1. 

C.5.2  Healthy  Soldier 

When  the  combat  effective  soldier  has  a  fully  functioning  CSC  system  and  the  CSC  does 
not  detect  any  elevated  cognitive  workload  states,  then  no  alert  is  generated.  In  this  case, 
the  default  soldier  symbol  is  used,  without  modifiers.  It  is  anticipated  that  this  will  be  the 
most  frequently  used  symbol. 

C.5.3  Cognitive  State  Alerts 

In  monitoring  cognitive  state,  the  CSC  system  will  continually  assess  the  soldier’s 
physiology  (ECG)  and  neurophysiology  (EEG).  Computer  algorithms  will  process  the 
data  to  determine: 

(a)  whether  soldier  is  operating  under  increased  cognitive  workload  and 

(b)  whether  the  CSC  system  itself  is  working  properly. 


If  the  CSC  system  is  working  properly  and  it  detects  that  the  soldier  is  operating  under 
increased  workload,  the  computer  algorithms  will  generate  and  transmit  the  appropriate 
status  alert  (i.e.,  Medium  or  High  workload). 
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The  modifier  for  the  appropriate  CSC  status  alert  will  be  appended  to  the  lower  right 
comer  as  the  “K”  modifier.  The  “K”  modifier  is  designated  for  Combat  Effectiveness 
([2525B],  Table  IV,  page  52).  Form,  length,  and  coloring  will  follow  MIL-STD-2525B. 
The  modifier  is  placed  in  a  small  text  box  to  allow  background  color  coding  to  emphasize 
the  distinction  between  the  levels  of  the  CSC  status  alerts. 

C.5.4  System  Fault  Alert 

If  at  any  point  the  soldier’s  CSC  system  itself  is  not  functioning  properly,  a  modifier  for 
the  modifier  for  the  CSC  system  fault  alert  will  be  appended  to  the  symbol  on  the  middle 
of  the  right  side  as  the  “H”  modifier.  The  “H”  modifier  is  designated  for  Additional 
Information  ([2525B],  Table  IV,  page  52).  This  alert  needs  to  be  distinct  from  other 
defined  alerts. 

C.5.5  Unit  Roll-Up 

For  the  FFW  OTM  2007,  there  will  be  no  unit  roll-up  since  only  one  soldier  will  be 
equipped  with  the  CSC  system. 

C.5.6  Final  Composite  Symbols 

Table  C-2  illustrates  the  tactical  symbols  for  cognitive  state  feedback. 


Table  C-2.  Tactical  symbols  for  cognitive  state  feedback. 


State 

Cognitive  State: 
Medium  Workload 

Cognitive  State: 
High  Workload 

CSC 

System  Fault 
Alert 

Symbol 

(J) 

xn^_^4medwl| 

Xc 

HI  WL  | 

6 

= - 1 SYST  | 

C.6  System  Logic 

Figure  C-2  illustrates  the  system  logic  of  the  CSC  system. 
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Figure  C-2.  CSC  system  logic. 


